R Tools for Data Science

Not only is R the very best language for statistical inference, it is rapidly evolving to accommodate the needs of data scientists. In this session, we will discuss some of the tools available in R for practicing data science. We will begin with a review some of the fundamental strengths of the R language and then discuss how R is integrating into the world of production level data science.

For the fourth year in a row, R has made it into the top 10 list of the IEEE Spectrum Magazine’s survey of general purpose programming languages. That R should be there at all is astounding! Much of R’s popularity can be attributed to the growth of data science.

A recent StackOverflow Post contained the following chart which shows the top twelve industries in the United States and the United Kingdom that generated R related questions.

The chart clearly shows that after Academia, R traffic originating from the Healthcare dominates the traffic by generated by other industries.

Moreover, another chart published in the same post shows R traffic is generally growing faster in industries where it was already popular. Academia and Healthcare are pulling away from the pack.

In a follow-up Stackoverflow post R was identified as the least disliked language.

Our Agenda

After a short discussion of why the R Language itself is particularly well suited to data science, we will look at examples of R tools in seven categories:
* Working in R
* Manipulating Data
* Accessing Data
* Visualizing Data
* Predictive Modeling
* Machine Learning with Big Data
* Sharing Results

Agenda Details
* The R Language itself (10 min)
* A very brief history
* The Structure of R
* The R Package System
* Tools for Working in R (5 min)
* R Markdown
* R Notebooks
* Tools for Manipulating Data (20 min)
* The Tidyverse
* Tools for Accessing Data (5 min)
* Accessing Databases
* Tools for Visualizing Data (5 min)
* htmlwidgets
* Trelliscope
* Predictive Modeling Tools (10 min)
* The caret package
* Tools for Big Data Platforms (30 min)
* Spark and Sparklyr
* Keras and Tensor Flow
* Tools for Sharing Results (5 min)
* Shiny

________________________________________________________________________________________

The R Language Itself

A Very Brief History of R

To appreciate this R’s growth, consider that R evolved from a relatively modest effort to develop an interface to Fortran statistical routines to a full featured language that provides access to the world’s richest repository of statistical and machine learning algorithms. The following dates outline a short history of how this happened.

R is a direct descendant of the S Language developed at Bell Labs by John Chambers, Rick Becker and other, but it was also heavily influenced by Lisp and Scheme. Robert Gentleman and Ross Ihaka introduced Lisp like semantics to an ideas such as lexical scoping.

________________________________________________________________________________________

The Structure of the R Language

R is all about “Flow”

The R language was designed to enhance the experience of data exploration and statistical inference. The original intent of the S designers to make a convenient interface for their high powered Fortran rourines evolved into a project to make R a full-featured language for statistical inference and modeling. People productivity is at the core of the R experience. As John Chambers puts it in his book Extending R, “One of the attractions of R has always been the ability to compute an interesting result quickly.” Ease of use and the ability to stay in the “flow” of an analysis is more important than computational efficiency.

R is a functional, object-based language. The three basic principles underlying R are:
* Everything that exists in R is an object
* Everything that happens in R is a function call
* Interfaces to other software are part of R

Chambers elaborates on this last point as follows: “A key motivation for the original S remains important now: to give easy access to the best computations for understanding data.” We see several examples of the efforts R developers put into connecting to the best things out there later in this presentation.

Objects and Functions

The interplay of objects and functions in R is apparent in the way in which data and statistical models are packaged. For example consider a simple linear model:

x <- rnorm(100)
y <- rnorm(100)
reg <- lm(y ~ x)
summary(reg)

Call:
lm(formula = y ~ x)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.7082 -0.7468  0.2222  0.7854  2.2538 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.08342    0.11116  -0.750    0.455
x            0.13427    0.11811   1.137    0.258

Residual standard error: 1.111 on 98 degrees of freedom
Multiple R-squared:  0.01301,   Adjusted R-squared:  0.002943 
F-statistic: 1.292 on 1 and 98 DF,  p-value: 0.2584

As the summary above indicates, R only returns model outputs when you ask for them and methods for generating summaries tend to be parsimonious. The model object reg packages quite a bit of information about the model.

str(reg,give.attr=FALSE)

List of 12
 $ coefficients : Named num [1:2] -0.0834 0.1343
 $ residuals    : Named num [1:100] 1.461 0.334 -1.726 0.842 0.874 ...
 $ effects      : Named num [1:100] 0.88 -1.263 -1.854 0.723 0.718 ...
 $ rank         : int 2
 $ fitted.values: Named num [1:100] -0.0796 0.0257 -0.1211 -0.186 0.0674 ...
 $ assign       : int [1:2] 0 1
 $ qr           :List of 5
  ..$ qr   : num [1:100, 1:2] -10 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 ...
  ..$ qraux: num [1:2] 1.1 1.09
  ..$ pivot: int [1:2] 1 2
  ..$ tol  : num 1e-07
  ..$ rank : int 2
 $ df.residual  : int 98
 $ xlevels      : Named list()
 $ call         : language lm(formula = y ~ x)
 $ terms        :Classes 'terms', 'formula'  language y ~ x
 $ model        :'data.frame':  100 obs. of  2 variables:
  ..$ y: num [1:100] 1.381 0.36 -1.847 0.656 0.941 ...
  ..$ x: num [1:100] 0.0282 0.8127 -0.2805 -0.7643 1.1236 ...

Functions in R can call other functions. This feature is well adapted to writing functions to estimate maximum likelihood and other statistical algorithms.

# Two different rounding options
round(pi,4); signif(pi,4)     

[1] 3.1416
[1] 3.142

# This is the way to have one function call another function
jmean <- function(x,FUN,...){
               m <- FUN(sum(x)/length(x),...)
               return(m)}
x <- rnorm(100)
jmean(x,round,4); jmean(x, signif,4)

[1] -0.0278
[1] -0.02777

Missing Values

A significant advantage that R has over other scripting languages is that it has an innate mechanism for dealing with missing values: NA.

z <- c(1:3, NA)
z

[1]  1  2  3 NA

is.na(z)

[1] FALSE FALSE FALSE  TRUE

It is easy to identify and work with missing values in R.

a <- 1:5
b <- rnorm(5)
c <- LETTERS[1:5]
dF <- data.frame(a,b,c)
dF$b[3] <- NA; dF$c[4] <- NA
dF

na.omit(dF)

R also has several packages devoted to missing value imputation including: Amelia, BaBooN, cat, ForImp, Hmisc, impute, imputeMDR, kmi, mi, mice, MImix, missForest, MissingDataGui, missMDA, mitools, mix, mtsdi, norm, pan, robCompositions, rrcovNA, sbgcop, VIM, yaImpute, Zelig

The Data Frame

A data frame is a list in which each row may represent an observation and each column a variable, is a natural data structure for statistical analysis. This consistent, ubiquitous data structure provides R with a big technical advantage. This is no accident. Both Robert Gentleman and Ross Ihaka believed in Niklaus Wirth’s dictum: algorithms + data structures = programs.

________________________________________________________________________________________

The R Package System

There are over 11,700 packages on CRAN, R’s central repository. The algorithms in these packages comprise a transparent, documented statistical resource of great value. Searching the through CRAN is a challenge, but the CRAN Task Views lists of R packages, curated by experts and organized by application area mitigate the problem.

________________________________________________________________________________________

Tools for Working in R

A good place to start is with an IDE and R Markdown Let’s take a few minutes to examine this notebook and the R Markdown code.

________________________________________________________________________________________

Tools for Manipulating Data

The Tidyverse

Another great advantage of R is its ability to facilitate the construction of Domain Specific Languages. As Shiny’s creator Joe Cheng puts it (Interiew with Joe Cheng):

If you look beyond the syntax, R really is conceptually very much like Lisp in a lot of ways. One of those ways is that it makes it very, very easy to compute on the programming language itself. …

I think day-to-day, R programmers probably don’t think about these things, but the elegant, terse syntax of dplyr and the pipe operator are possible because of how malleable a language R is and how great it is for writing DSLs in it.

Personally, one of my pet peeves during these language wars is when people say that one of the dierences between say Python or Julia and R is that R is a DSL for stats, whereas these other things are general purpose languages. R is not a DSL. It’s a language for writing DSLs, which is something that’s altogether more powerful. I actually think that Julia has many of these same characteristics, but Python, even though it obviously has its own strengths, certainly doesn’t share that same level of fexibility.

One way to think of the Tidyverse is that it is a DSL for doing data science. The packages that comprise the tidyverse offer a consistent, integrated set of functions for implementing the canonical data science workflow.

The name “tidyverse” comes from the concept of “tidy” data which Hadley Wickham, the tidyvrese’s principal architect and implementer, describes as follows:

Tidy data is a standard way of mapping the meaning of a dataset to its structure. A dataset is messy or tidy depending on how rows, columns and tables are matched up with observations, variables and types. In tidy data: (1) Each variable forms a column, (2) Each observation forms a row and (3) Each type of observational unit forms a table.

Manipulating and Transforming Data

The dplyr package which provides a consistent set of verbs for manipulating data provides the backbone structure for the import, tydy and transform sequence of the data science workflow. The basic dplyr verbs are: * filter() to select cases based on their values
* arrange() to reorder the cases
* select() and rename() to select variables based on their names
* mutate() and transmute() to add new variables that are functions of existing variables
* summarise() to condense multiple values to a single value
* sample_n() and sample_frac() to take random samples

Here, we will look at a few examples of working with these verbs using nycflights13::flights, a dataset containing all 336776 flights that departed from New York City in 2013. The data comes from the US Bureau of Transportation Statistics, and is documented in ?nycflights13

#install.packages("tidyverse","nycflights13")
library(nycflights13)
library(tidyverse)
data(flights)
flights

In this first example, we use dplyr::filter() to fetch flights that arrived early in January.

# filter by departure delay and print the first few records
flights %>% filter(month == 1, dep_delay < 0)

arrange() reorders rows. It takes a data frame, and a set of column names (or more complicated expressions) to order by. If you provide more than one column name, each additional column will be used to break ties in the values of preceding columns:

arrange(flights, sched_dep_time, dep_delay)

select() allows you to select columns by name. Here we select the columns origin, dest and air_time, and then sort by air_time from largest to smallest.

select(flights, origin, dest, air_time) %>% arrange(desc(air_time))

mutate() allows you to add a new column to a data frame. Here we add in a column for airspeed.

mutate(flights,
  gain = arr_delay - dep_delay,
  speed = distance / air_time * 60
)

summarise() will compute a statistic and collapse a data frame into a single row.

dplyr::summarise(flights,
  delay = mean(dep_delay, na.rm = TRUE)
)

The group_by() functions allows you use the verbs discussed above on groups of observations in a dataset. Here we compute mean distance and arrival delay for each individual plane and plot the results.

by_tailnum <- group_by(flights, tailnum)
delay <- dplyr::summarise(by_tailnum,
  count = n(),
  dist = mean(distance, na.rm = TRUE),
  delay = mean(arr_delay, na.rm = TRUE))
delay <- filter(delay, count > 20, dist < 2000)
# plot delays
library(ggplot2)
ggplot(delay, aes(dist, delay)) +
  geom_point(aes(size = count), alpha = 1/2) +
  geom_smooth()  +
  scale_size_area(max_size = 4)

For a more complete presentation using this data see the dplyr vignette.

________________________________________________________________________________________

Tools for Accessing Data

Accessing Databases

With the aid of the DBI package, dply is able to import data from several different open source data bases including MySQL and MariaDB with the RMariaDB package, Postgres and Redshift with RPostgressSQL, and SQLite with RSQLite.

The following simple example illustrates working with SQLite. Again, we use the nycflights13 data set to create a SQLite table with 1,000 rows and 19 columns.

library(tidyverse)
library(DBI)
library(RSQLite)
library(dbplyr)

Attaching package: ‘dbplyr’

The following objects are masked from ‘package:dplyr’:

    ident, sql

library(nycflights13)
con <- DBI::dbConnect(RSQLite::SQLite(), path = ":memory:")
copy_to(con, nycflights13::flights, "flights",
  temporary = FALSE, 
  indexes = list(
    c("year", "month", "day"), 
    "carrier", 
    "tailnum",
    "dest"
  )
)
flights_db <- tbl(con, "flights")
flights_db 

Now that we have established a connection to the database table, we can use the normal dplyr verbs to query the data.

# List departure delay and arrival delay for year month and day.
flights_db %>% select(year:day, dep_delay, arr_delay)

# Find records with departure delay greater than 240 minutes.
flights_db %>% filter(dep_delay > 240)

# Summarize mean departure delay by destination airport
flights_db %>% 
  group_by(dest) %>%
  dplyr::summarise(delay = mean(dep_delay))

# disconnect from database
dbDisconnect(con)

dplyr can also be use with data stored in commercial databases. The bigquery enables working with data in Google’s BigQuery platform, and the odbc permits many commercial databases to be used as a dplyr backend through the open database connectivity protocol,

For additional information on basic dbplyr functionality see the vignette. For an advanced treatment using dplyr with commercial grade databases see the three blog posts from Edgar Ruiz:
* Databases using R
* Visualizations with R and Databases * Database Queries with R * Enterprise-ready dashboards with Shiny and databases.

Be sure to catch Edgar’s talk “Databases Using R” 11:30 AM tomorrow in room TR2

________________________________________________________________________________________

Tools for Visualizing Data

Javascript Visualizations with htmlwidgets

There are three major plotting systems in R: (1) base graphics, (2) lattice graphics and (3) ggplot2, an example of which we have seen above. Additionally, there is a significant amount of development work going on to allow R users to produce JavaScript visualizations directly from R. In this section we will see an example of working with R’s htmlwidgets library.

For this example we will work with the htmlwidget dygraphs which is useful for producing interactive time series visualizations and use it to visualize the monthly flow from the Nile River from 1871 through 1984.

library(pracma)             # for Nile river data
data(nile)
head(nile)

This data set is not tidy. For each year, the information about monthly flow is spread over several columns. To tidy things up, we need to reshape the data frame into “long form” where the columns comprise the variables Year, Month and Flow. In the code below, the tidy function gather() accomplishes this. arrange() sorts the data by Year and Month, and mutate adds a new date variable to the data frame.

nile_long <- nile %>% 
                  gather(Jan:Dec, key = "Month", value = "Flow") %>% 
                  arrange(Year,match(Month,month.abb)) %>% 
                  dplyr::mutate(
                  Date = as.Date(paste(Month,"-","15","-",as.character(Year),sep=""), format="%b-%d-%Y"))
  
head(nile_long)

Next, we produce some exploratory plots with ggplot2

# Plot the time series
p <- ggplot(nile_long[100:300,],aes(x=Date,y=Flow))
p + geom_line() + geom_point(shape=1,col="red") + 
  ylab("Flow in cubic meters / second") + 
  ggtitle("Monthly Flow of Nile River at Dongola Station")

# Boxplots of monthly flows
b <- ggplot(nile_long,aes(factor(Month),Flow))
b + geom_boxplot() +
  xlab("Month") +
  ylab("Flow in cubic meters / second") + 
  ggtitle("Variation of Flow at Dongola Station by Month")

We finish up by creating an interactive, JavaScript D3 interactive plot. To get the data ready for the dygraph() function we must make it an xts time series object.

# Create an interactive graph with a Javascript library
library(xts)
library(dygraphs)
# Make into a time series object
nile_ts <- xts(nile_long$Flow,
               order.by=nile_long$Date,
               frequency=12,start=c(1871,1))
# Plot wit htmlwidget dygraph
dygraph(nile_ts,ylab="cubic m / s", 
        main="Nile Monthly Flow Data") %>%
  dySeries("V1",label="Flow") %>%
  dyRangeSelector(dateWindow = c("1871-01-01","1984-12-01"))

To see some spectacular visualizations have a look at the htmlwidgets for R gallery.

________________________________________________________________________________________

Visualizing Large Data Sets with Trelliscope

Trelliscope is a system for producing detailed visualizations of very large, complex datasets. Thousands of plots are produced and the system computes metrics on each plot that describe interesting quantitative and qualitative features. Users can then sample, filter, or sort the plots based on the metrics. In Trelliscope terminology, these metrics are called “cognostics”, a term coined by John Tukey which stands for “computer guiding diagnostics”

Example using gapminder life expectancy data

Trelliscope is part of the DeltaRho Project. Look here for a paper that summarizes the system.

________________________________________________________________________________________

Predictive Modeling Tools

See the separate caret notebook.

________________________________________________________________________________________

Tools for Big Data Platforms

See the notebooks: * R_Keras_TensorFlow * sparlkyr

________________________________________________________________________________________

Tools for Sharing Results

Shiny

Shiny is an open source R package that provides an elegant and powerful web framework for building web applications using R. Shiny helps you turn your analyses into interactive web applications without requiring HTML, CSS, or JavaScript knowledge. Shiny has become an essential communication platform for several enterprise data science companies.

Some Enterprise Level Shiny Applications:
* LRT Signal Analysis for a Drug from the FDA

• The SeaClass R Package from Seagate Technologies](https://rviews.rstudio.com/2017/10/23/the-seaclass-r-package/)
  

Some “How to” Shiny posts: * Enterprise-ready dashboards with Shiny and databases * Portfolio Volatility Shiny App * Printing From Flex Dashboard

To get started with Shiny: * The Shiny Showcase * Learn Shiny * Shiny Gallery